Page 2 of 44
Knowledge Distillation (KD)
The generic framework of teacher-student knowledge distillation training. (Image source: Gou et al. 2020)
Goal: Transfer knowledge from expensive teacher model(s) into a smaller* student model,
while retaining capabilities.
*Self-distillation uses the same-sized student and teacher, and can still result in better performance.
Page 3 of 44
Distillation: Aren’t Bigger LLMs Always Better?
Page 4 of 44
Deployment of LLMs often
limited by their inference cost
or memory footprint
- Putting 100B parameters
on your smartphone
needs a lot of memory.
- You typically don’t want
to wait several minutes
for getting an output.
Distillation: Aren’t Bigger LLMs Always Better? No
Credit: This post by @swyx
LMSYS: Performance vs Model Pricing
Page 5 of 44
Distillation Recovers “Dark Knowledge”
See Dark Knowledge (Slides from Geoff Hinton)
Source: https://x.com/jxmnop/status/1877761437931581798
Page 6 of 44
This Tutorial:
Covers
Progress in
Distillation,
especially for
LLMs (Biased
towards
some of my
own work)
Page 7 of 44
Supervised Distillation (Hinton et al., 2015)
